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ABSTRACT 



r A system fortranscoding multiple channels of compressed 
i video streams jusing a self contained data^uriit such as" an 
autonomous frame includes an autonomous frame process- 
ing unit having an autonomous frame generator and an 
autonomous frame recoder. Hie autonomous frame genera- 
tor receives video data and divides it into a series of 
autonomous frames. Each autonomous frame preferably 
comprises 1) a frame header including all header informa- 
tion from the original video data plus enough additional 
information to allow the frame to be recoded using pre- 
defined autonomous frame structure, and 2) a frame pay load 
including the original video data information. The autono- 
mous frame recoder process the autonomous frames includ- 
ing extracting /processing parameters^extracting the video 
data and setting up or initializing the recoder to process the 
extracted video data. The autonomous frame recoder pref- 
erably further comprises a parser coupled to an initialization 
^unit and a recoder. The present invention also includes a 
method for processing video data including the steps o£ 
/receiving a video bitstream, /storing recoding informatio: 
^dividing the video bitstream 'into a plurality of autonomous 
frames each frame including a portion of the video bitstream 
and recoding information, outputting the plurality of autono- 
mous frames, receiving the plurality of autonomous frames, 
extracting processing information from the autonomous 
frame, extracting video data from the autonomous frame (J 
setting the recoding according to the processing informatio] 
■ and recoding the extracted video data; 

27 Claims, 13 Drawing Sheets 
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SYSTEM AND METHOD FOR 
TRANSCODING MULTIPLE CHANNELS OF 
COMPRESSED VIDEO STREAMS USING A 
SELF-CONTAINED DATA UNIT 

This Application claims the benefit of Ser. No. 60/094, 
364, filed Jul. 27, 1998. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to systems and 
methods for processing compressed bitstreams of data. In 
particular, the present invention relates to a system and a 
method for transcoding video bitstreams on frame and 
smaller basis. Stilt more particularly, the present invention 
relates to a system and method for transcoding multiple 
channels of compressed video streams using self-contained 
data units such as autonomous frames wherein such an 
autonomous frame includes a frame header portion and a 
frame payload portion. 

2. Description of the Related Art 

There are presently a variety of different communication 
channels for transmitting or transporting video data. For 
example, communication channels such as digital subscriber 
loop (DSL) access networks, ATM networks, satellite, or 
wireless digital transmission facilities are all well known. 
The present invention relates to such communication 
channels, and for the purposes of the present application a 
channel is defined broadly as a connection facility to convey 
properly formatted digital information from one point to 
another. A channel includes some or all of the following 
elements: 1) physical devices that generate and receive the 
signals (modulator/demodulator); 2) physical medium that 
carries the actual signals; 3) mathematical schemes used to 
encode and decode the signals; 4) proper communication 
protocols used to establish, maintain and manage the con- 
nection created by the channel. The concept of a channel 
includes but is not limited to physical channel, but also 
logical connections established on top of different network 
protocols, such as xDSL, ATM, wireless, HFC, coaxial 
cable, etc. 

The channel is used to transport a bitstream, or a con- 
tinuous sequence of binary bits used to digitally represent 
compressed video, audio or data. The bit rate is the number 
of bits per second that the channel is able to transport. The 
bit error rate is the statistical ratio between the number of 
bits in error due to transmission and the total number of bits 
transmitted. The channel capacity (or channel bandwidth) is 
the maximum bit rate at which a given channel can convey 
digital information with a bit error rate no more than a given 
value. A video channel or video program refers to one or 
more compressed bit streams that are used to represent the 
video signal and the associated audio signals. Also included 
in the video channel are relevant timing, multiplexing and 
system information necessary for a decoder to decode and 
correctly present the decoded video and audio signals to the 
viewer in a time continuous and synchronous manner. And 
finally, a multiplex is a scheme used to combine bit stream 
representations of different signals, such as audio, video, or 
data, into a single bit stream representation. 

One problem with existing communication channels is 
their ability to handle the transportation of video data. Video 
data is much larger than many other types of data, and 
therefore, requires much more bandwidth from the commu- 
nication channels. Since transmission of digitally sampled 
video data with existing communication channels would 



25 



require excessive amounts of time, compression is an 
approach that has been used to make digital video images 
more transportable. Digital video compression schemes 
allow digitized video frames to be represented digitally in 
5 much more efficient manner. Compression of digital video 
makes it practical to transmit the compressed signal by 
digital channels at a fraction of the bandwidth required to 
transmit the original signal without compression. Interna- 
tional standards have been created on video compression 
schemes and include MPEG-1, MPEG-2, H.261, H.262, 
H.263, etc. These standardized compression schemes mostly 
rely on several key algorithm schemes: motion compensated 
transform coding (for example, DCT transforms or wavelet/ 
sub-band transforms), quantization of the transform 
coefficients, and variable length encoding (VLC). The 
15 motion compensated encoding removes the temporally 
redundant information inherent in video sequences. The 
transform coding enables orthogonal spatial frequency rep- 
resentation of spatial domain video signals.' Quantization of 
the transformed coefficients reduces the number of levels 
io\ required to represent a given digitized video sample and is 
^the major factor in bit usage reduction in the compression } 
fprocess.JThe other factor contributing to the compression is 
the' use' of variable length coding (VLQ so that most 
frequently used symbols are represented by the shortest code 
word. In general, the number of bits used to represent a 
given image determines the quality of the decoded picture. 
The more bits used to represent a given image, the better the 
image quality. The system that is used to compress digitized 
video sequence using the above-described schemes is called 
an encoder or encoding system. 

In the prior art compression schemes, the quantization 
scheme is lossy, or irreversible process. Specifically, it 
results in loss of video textural information that cannot be 
recovered by further processing at a later stage. In addition, 
the quantization process has direct effect on the resulting bit 
usage and decoded video quality of the compressed bit 
stream. The schemes at which the quantization parameters 
are adjusted control the resulting bit rate of the compressed 
bit stream. The resulting bit stream can have either a 
constant bit rate (CBR) or a variable bit rate (VBR). CBR 
compressed bit stream can be transmitted over channel 
delivers digital information at a constant bit rate. 

A compressed video bit stream generally is intended for 
real-time decoded playback at a different time or location. 
The decoded real-time playback must be done at 30 frames 
per second for NTSC standard video and 25 frames per 
second for PAL standard video. This implies that all of the 
information required to represent a digital picture must be 
delivered to the destination in time for decoding and display 
in timely manner. Therefore, this requires that the channel 
must be capable of making such delivery. From a different 
perspective, the transmission channel imposes bit rate con- 
straint on the compressed bit stream. In general, the quan- 
tization in the encoding process is adjusted so that the 
resulting bit rate can be accepted by the transmission chan- . 
nel. 

Because both temporal and spatial redundancies are 
removed by the compression schemes and because of vari- 
able length encoding, the resulting bit stream is much more 
sensitive to bit errors or bit losses in the transmission process 
than if the uncompressed video is transmitted. In other 
words, minor bit error or loss of data in compressed bit 
stream typically results in major loss of video quality or even 
complete shutdown of operation of the digital receiver/ 
decoder. 

Further, a real-time multimedia bit stream is highly sen- 
sitive to delays. A compressed video bit stream, when 
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transmitted under excessive and jittery delays, will cause the bitstream, storing recoding information, dividing the video 

real-time decoder buffer to under flow or overflow, causing bitstream into a plurality of autonomous frames each frame 

the decoded video sequence to be jerky, or causing the audio including a portion of the video bitstream and recoding 

video signals out of synchronization. Another consequence information, outputting the plurality of autonomous frames, 

of the real-time nature of compressed video decoding is that s receiving the plurality of autonomous frames, extracting 

lost compressed data will not be re-transmitted. processing information from the autonomous frame, extract- 

Despite the increase in channel bandwidth, there contin- ing video data from the autonomous frame, setting the 

ues to be a need for adjusting the number of bits for recoding according to the processing information and recod- 

representing a bitstream to the amount of available channel ing the extracted video data, 

bandwidth. Another particular problem, especially when 10 

several channels are multiplexed over a single channel, is the BRIEF DESCRIPTION OF THE DRAWINGS 

allocation of the available bandwidth to multiple channels. rg. 1 is a block diagram of a transcoding system 

Often it is necessary to recode bitetreams to maximize the inchlding the aut onomous frame processing unit of the 

utilization of the channel bandwidth. However, the use of present invention 

compression techniques also introduces significant compu- 15 # * 4 

tational complexity into both the encoding and decoding FI , G 1 2 »» "ode diagram of a first and preferred embodi- 

process. Specifically, the compressed video bit streams, at menl J or autonomous frame processing unit of the 

any given bit rate, cannot be altered again to a different bit presen inven ion. 

rate without decoding and recoding. In addition, the result- FIG - 3 ^ a block diagram of a first embodiment for the 

ing number of bits required to represent digital video pic- 20 autonomous frame generator of the present invention, 

tures varies from picture to picture and the coded pictures FIG. 4 is a block diagram of a first embodiment for the 

are highly correlated as a result of motion estimation. The autonomous frame recoder of the present invention, 

problem of delivering real-time digital video bit stream over FIG. 5 is a block diagram of an exemplary embodiment 

a channel of a given bandwidth becomes even more complex for the recoder of FIG. 4. 

because the available bandwidth must be matched to the 25 FIQ fi fa a Wock ^ ^ of a &econd embodiment for me 

coded video bit stream rate. When the mismatch occurs, „ t c *u * • ** 

" ~~ " " , , ' autonomous frame processing unit of the present invention, 

recoding, or re-compression, must be done. A final problem _ . . . , r t - t r» 

is that existing recoding processes only allow recoding on a 7 1S * block of a third embodiment for the 

stream by stream basis. There are many instances when only autonomous frame processing umt of the present invention, 

a portion of a stream may need to be receded to resolve a 30 FIG. 8 is a block diagram illustrating several exemplary 

temporary shortage of channel bandwidth. embodiments for an autonomous frame based on an access 

Therefore, there is a need for a system and method for unit ( AU )> a s P Uce and a macro block - 

transcoding a bitstream on a frame or smaller basis. FIG. 9 is a block diagram of a fourth embodiment for the 

Furthermore, there is a need for a system that allows autonomous frame processing unit of the present invention, 

transcoding on a section of the compressed video data and 35 FIG. 10 is a block diagram of a fifth embodiment for the 

on an autonomous basis anywhere within a video stream. autonomous frame processing unit of the present invention. 

SUMMARY OF THE INVENTION PIG. 11 is a block diagram of a sixth embodiment for the 

# . „ tU * « . * autonomous frame processing unit of the present invention. 

The present invention overcomes the deficiencies and r & r 

limitations of the prior art with a system and a method for 40 FIG * 12 is a block dia g ram of a first statistical 
transcoding multiple channels of compressed video streams re-multiplexer using autonomous frame processing accord- 
busing ^autonomous frames/ More particularly, a system m g t0 the present invention. 

according to the present invention includes an autonomous FIG. 13 is a block diagram of a second statistical 
frame processing unit having an autonomous frame genera- re-multiplexer using autonomous frame processing accord- 
tor and an autonomous frame recoder. The autonomous 45 ing t0 the present invention. 

frame generator receives video data and divides it into a 

series of autonomous frames. Each autonomous frame pref- °™™c R nI I nL 0 ^ E 

erably comprises 1) a frame header including all header PREFERRED EMBODIMENTS 

information from the original video data plus enough addi- While the present invention will now be described with 

tionai information to allow the frame to be recoded using 50 particularity for the handling of MPEG-2) digital video 

pre-defined autonomous frame structure, and 2) a frame compression, those skilled in the art will recognize that the 

pay load including the original video data information. The principles of the present invention may be applied to a 

autonomous frame generator outputs autonomous frames to variety of other related video compression schemes such as 

the autonomous frame recoder which in turn process the the H.26X video conference signals. Specifically, the present 

autonomous frame including extracting processing 55 applications discloses a technique of adding and processing 

parameters, extracting the video data and setting up or autonomous frames that can be used to selectively perform 

initializing the recoder to process the extracted video data. recoding. Furthermore, those skilled in the art will recognize 

The autonomous frame recoder preferably further comprises that the bit streams discussed below arc unscrambled. This 

a parser coupled to an initialization unit and a recoder. The allows direct access to all the bitstream content for rate 

autonomous frame recoder outputs a bitstream that has been 60 reduction, recoding, and timing re-stamping. In cases where 

adjusted in the number of bits used to represent the data. In the bitstreams are scrambled, they first need to be 

other embodiments, the autonomous frame processing unit unscrambled, spliced and then re -scrambled, 

may include a plurality of autonomous frame generators and System Overview 

a plurality of autonomous frame recoders coupled in a Referring now to FIG. 1, a preferred embodiment of a 

variety of different configurations. 65 system 100 constructed according to the present invention 

The present invention also includes a method for process- and including an autonomous frame processing unit 108 is 

ing video data including the steps of: receiving a video shown. The system 100 is adapted for use on an MPEG-2 
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transport stream. The preferred embodiment of the system the following sections, the present invention is disclosed in 

100 preferably comprises a separation unit 102, a the context of reooding of MPEG-2 bit streams, whether it 

de-multiplexer 104, an elementary stream bit parser 106, the is transport stream, program stream, PES stream or elemen- 

autonomous frame processing unit 108, a stream bit multi- tary stream. This section provides brief overview of -the 

plexer 114, and a transport multiplexer 116. 5 MPEG-2 bit stream syntax for convenience and ease of 

The separation unit 102 has an input coupled to line 120 understanding of the concept in the present invention, 
and receives an MPEG-2 transport stream. The separation MPEG-2 compression standard consists of two layers of 
unit 102 preferably de-multiplexes the system layer stream, information: the video or audio layer and the system layer, 
removing filler packets/frames as appropriate, to obtain the The video layer includes the elementary stream (ES) layer, 
video bit stream, the audio bit stream and a data bit stream. 10 This layer defines how compressed video (or audio) signals 
The video bit stream is provided on a first output, the audio are sampled, motion compensated, transform coded, 
bit stream is provided on a second output, and a data bit quantized, and represented by different variable length cod- 
stream is provided on a third output. Those skilled in the art ing (VLQ tables. The recoding (or transcoding) of pre- 
will understand the operation of the separation unit 102, compressed MPEG-2 bit streams is a process in which the 
based on the protocol being used for the bit stream. Based 15 bit stream signal is redefined in this layer, 
on the protocol, the bit stream can be divided into the fh e system layer is defined to allow the MPEG-2 decoder 
sub-streams according to the present invention. The audio t 0 correctly decode audio and video signals, present the 
stream and data stream are output directly to the transport decoded result to the video screen in a time continuous 
multiplexer 116 which recombines these streams with the manner. The system layer also includes provisions that allow 
receded or processed video. Even though a compressed bit 20 unambiguous multiplexing and separate of audio and video 
stream typically includes a multiplex of compressed audio, compressed signals, and different channels of audio and 
video and auxiliary data bit streams, recoding process in the video compressed signals. The system layer consists of two 
present invention refers only to the process applied to sub-layers. The first layer is the PES layer, this layer defines 
compressed video bit stream. how the ES layer bit stream is encapsulated into variable 

The de-multiplexer 104 is coupled to receive the video 25 length packets, called PES packets. In addition, presentation 

stream from the separation unit 102. The de-multiplexer 104 aD( j decoding time stamps (PTS/DTS) are added to the PES 

extracts a video^elementary stream payload from the video packets. There are two different sub-layers above the PES 

transport stream and in turn sends the video elementary layer, the transport layer and program system layer, 

stream payload to the elementary stream bit parser 106. The ^ M j defines how the pES ke(s afe 

elementary stream bit parser 106 receives the output of the 30 cketized mt0 ^ sized , rt ket of 188 

de-multiplexer 104 and divides it into access units that can bytes . Additional timing information and multiplexing infor- 

mclude one or more of a transform coefficient component, a mation afc added tQ me , r rt la ^ resuhi stream 

motion vector component, and an auxUiary mformauon of Qrt keU ^ calIed h rt stream T ort 

component. The access units are provided on line 122 and s , ream fa optimized for ^ m environments where err or S are 

sent to the autonomous frame processing unit 108. 35 ^ ^ u s , or taataMoa m lossy or noisy 

TheautonomousframeprocessingumtlOSpreferablyhas media T kal of ti n stream inckdes 
an input and output. The input of the autonomous frame Direct Broadcast Service (DBS) digital or cable 
processing unit 108 is coupled to the output of the elemen- servkes> broadband transmission systems, etc. 
tary stream bit parser 106 to receive access units. The m , nr , n 
autonomous frame processing unit 108 receives video data 40 ™ e P^gram system layer defines how the PES packets 
and divides it into a series of autonomous frames and are encapsulated into variable size packets^ Additional Urn- 
performs recoding on the autonomous frames. Each autono- m S and mulUplexing information are added to the program 
mous frame preferably comprises 1) a frame header includ- system layer. The program stream is designed for use m 
ing picture header information from the original video data relaUvely error-free environments and is suitable for apph- 
plus enough additional information to allow the frame to be 45 ca ' l0ns that mav mvolve software processing of system 
recoded, and 2) a frame payload including the original video information such as mteractive multimedia apphcaUons^A 
data picture information. Hie autonomous frame processing ^ *l &p ? Uc *™ of P ro ^ am strcam mcludcs Dl ® ia] Vcr ' 
unit 108 recodes the video data on an autonomous frame to satlle Dlsks < DVD > Vlde0 servers '. 
generate and video transport stream. The output of the In general a video bit stream can be in elementary stream 
autonomous frame processing unit 108 is coupled by line 50 (ES) format, which means that no PES, transport or program 
124 to the stream bit multiplexer 114. svslem laver information is added to the bit stream. The 

The stream bit multiplexer 114 essentially performs an video bit stream can also be represented in the form of PES 

inverse function of the de-multiplexer 104, and combines s^am, transport stream or program stream. For a given 

the output of the autonomous frame processing unit 108 with video bit stream, the difference between these different bit 

other information to re-form the video bitstream. The output 55 stream represented in the different layers lies in the timing 

of the stream bit multiplexer 114 is coupled to an input of the information, multiplexing information and other informa- 

transport multiplexer 116 along with the audio and data lion not directly related to the recoding process. The infor- 

signals from the separation unit 102. These signals arc then mation required to perform recoding, however, is contained 

combined by the transport multiplexer 116 and output on in the elementary stream layer. 

line 126. Those skilled in the art will recognize from FIG. 1 60 The discussion below on recoding is, therefore, not lim- 

that the present invention can be applied to bit stream based ited to bit streams in any one of the layers. In other words, 

on other transform schemes. the discussion on how to recoding bit streams in one layer, 

MPEG-2 Bit Stream Syntax say in elementary stream layer, can be straightforwardly 

As has been noted above, the present invention is extended to PES stream, transport stream or program 

described in detail with reference to the syntax used in 65 streams as well, by properly taking additional layering 

MPEG-2, but the concept of an autonomous frame applies to information into considerations, even though the elementary 

any number of video transport and compression schemes. In stream layer is the focus as shown in FIG. 1. 
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Autonomous Frame Data Structure 

The present invention advantageously creates a new con- 
cept of an autonomous frame data structure. The autono- 
mous frame is a scheme for partitioning the compressed 
video bit stream such that each unit or autonomous frame, as 
a result of the partitioning, forms a self-contained data unit 
that can be recoded. Below the partitioning of the coded 
video bit stream is denned to achieve this object. While the 
discussion focuses on the coding of MPEG-2 elementary 
stream, the "autonomous frame" concept described here can 
be applied to other compressed video bit streams as well. 

An autonomous frame is a data unit constructed based on 
a subsection of the compressed video bitstream that contains 
all of the information necessary to recode this data unit 
without using any additional information. Specifically, the 
data unit contains all the information required for the 
recoder to initialize itself and operate on the data unit. An 
autonomous frame is defined as consisting of the following 
two key components: a frame header (AFH) and frame 
payload (AFP). The autonomous frame is preferably derived 
from a subsection of the original compressed video 
bitstream, includes an associated receding method, and has 
the information contained in the frame header of an autono- 
mous frame sufficient to initialize the recoder to perform 
autonomous recoding. It is important to point out that a 
recoding scheme must be defined together with the autono- 
mous frame data structure. For example, an autonomous 
frame, when used as input to a recoder which is not defined 
with the autonomous frame will not necessarily make the 
recoding process autonomous, and vice versa. In other 
words, the recoding method and the type in the autonomous 
frame are coupled and must match for recoding to be 
properly performed. 

Referring now to FIG. 8, one embodiment 802 for the 
autonomous frame adapted for use in processing an MPEG2 
video bit stream is shown. This autonomous frame imple- 
mentation is defined as a data unit of coded picture without 
using the data from other coded pictures. In addition, 
requantization is assumed to be the method of re-encoding. 
For MPEG-2 video bit stream, each autonomous frame is 
advantageously constructed around the boundary of the so 
called access unit (AU) defined according to the MPEG-2 
standard. For MPEG-2, the present invention defines an 
access unit (AU) as all the coded data for a picture, and any 
stuffing that follows it, up to but not including the start of the 
next access unit. If a picture is not preceded by a group_ 
start_code or a sequence _Jicader_code, the access unit 
begins with the picture start code or picture_header. If a 
picture is preceded by a group_start_code and/or a 
sequence_header_code, the access unit begins with the first 
byte of the first of these start codes. If it is the last picture 
preceding a sequence_end_code in the bit stream all bytes 
between the last byte of the coded picture and the sequence.. 
end_code (including the sequence_end_code) belong to 
the access unit. 

In terms of the syntax of MPEG-2 video bit stream, an 
access unit (AU) is defined in the video elementary stream 
layer 810 and is shown in the following table. 

TABLE 1 



TABLE 1 -continued 



Access Unit Header (AUH) 



sequencc_header 
sequence_extens ion 
sequence_user_data 
OOP_header 
user_data 
picture_header 
picture_eJttension 
extensio a_user_data 



Optional 



Optional 



Access Unit Payload (AUP) 
Picture Data 



slice 0x101 



slice 0x1 XX 
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20 



However, it is straightforward to map the access unit infor- 
mation to the upper layer packets, such as PES or transport 
packets. There may be cases when a transport or PES packet 
contains bits from two or more access units, in which case 
only the bits of the access unit under discussion is consid- 
ered and all bits belonging to other access units are ignored. 
For convenience all data bits in an access unit immediately 
before the slice start code 0x101 but after the end of the 
picture data are defined as access unit header (AUH) bits. 
The rest of the bits of an access unit are defined to be the 
access unit payload (AUP) bits. The AUH contains infor- 
mation that is used to assist the decoding process and is 
maintained until being updated by the AUH contents from 
the next access unit. Some of the information bits in AUH 
must be used to perform the recoding process and thus must 
be available to the recoder. 

For MPEG-2, the present invention defines the self con- 
tained data structure, called an autonomous frame, to include 
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a frame header (AFH) and frame payload (AFP), as noted 
above. fThe^t6nb^ous"frame header (AFH) includes an 1 
updated version of AUH and additional auxiliary informa- 
tion used to assist the recoding process. For example, such, 
^auxiliary information preferably includes a bit budget per 
- picture (target bit usage of x bits), presentation:and display 
time stamps^ length of the frame in byte7)r packet/etc. The 
auxiliary information bits include all-the~~ information to 
perform recoding of the access unit. In other words, the AFH 
can be considered list of all global variables and their values 
that may be needed for recoding. Again, it is important to 
point out that the particular recoding technique.described in 
FIG. 4 for recoder 408 is assumed to be^requantization. This 
approach advantageously does not require tfre^useoTrefer- 
ence coded pictures. 

The autonomous frame payload (AFP) is preferably the 
same as the AUP, but may be different. The AUP does not 
exclude the scenario when all elementary stream bits are 
contained as part of the PES or transport packets. In this 
case, all information contained in the transport packet header 
fields are not considered part of AUP. It is important to point 
out that AFP described above may be interpreted as other 
than a copy of the AUP. When the access unit is in pack- 
etized form of the upper layer syntax, the associated upper 
layer system information may also be carried forward into 
the autonomous frame payload. To make the concept easier 
to explain, we limit our discussion to the access unit in its 
elementary stream form, and the assumption that the AUP is 
the same as the AFP. 

The relationship of the autonomous frame to the access 
unit is best illustrated in FIG. 8. In order the construct the 
autonomous frame 802, a parsing process is used to obtain 
the frame header. Specifically, the auxiliary information 
contained in the frame header is derived not from the latest 
access unit only, but from all of the access unit headers of 
all previous access units, up to and including the current 
access unit. A processing unit, called the pre-parser, must be 
used to generate the autonomous frame as will be discussed 
below. The pre-parser extracts the AUH information col- 
lected so far from all of the access units and uses it to update 
and set the proper fields in the auxiliary information data 
section of the autonomous frame. Once the AFH is created 
the AFP which is the video data from the original bit stream 
is added to create the autonomous frame 802. 



11/19/2003, EAST Version: 1.4.1 



US 6,483,543 Bl 



10 



10 



15 



Those skilled in the art will recognize that the autono- 
mou^fxame concept is not limited to a mapping between 
access units^and autonomous frames, but could occur at a 
variety of other levels to provide any level of partitioning of 
a video bitstream. Two alternate embodiments 804, 806 for 
the autonomous framearc also shown in FIG. 8. The other 
embodiments 804, 806 of the autonomous frame define the 
frame boundary to coincide with smaller data units within an 
access unit. For example, the autonomous frame 804 has an 
AFH that includes the auxiliary^information, but information 
from the AUH, and the slice header as graphically depicted. 
r The AFP for the autonomous frame 804 includes only a 
portion of the data from the AUP, namely, the slice pay load. 
A similar embodiment on a finer level of granularity is 
shown as autonomous frame 806 where the AFH includes 
-- the auxiliary information, information from the AUH; the 
slice header and the macro block header, and4he x AFP 
includes the macro block payload. The autonomous frame^ 
may„ be alternatively defined to be one autonomous frame \ 
/'per multiple macro blocks,/one autonomous frame for any s 20 
-contiguous section of a coded picture, or a variety of other v 
embodiments for any desired level of granularity, along with 
the associated appropriate recoding scheme. 
■ Autonomous Frame Processing Unit 

Referring now to FIG. 2, the first and preferred embodi- 
ment for the autonomous frame processing unit 108a of the 
present invention is shown in more detail. The autonomous 
frame processing unit 108opreferably comprises an autono- 
mous frame generator 202 and an autonomous frame recoder 
204, The autonomous frame generator 202 receives video 
data and divides it into a series of autonomous frames 802 
as described above with reference to FIG. 8. The autono- 
mous frame generator 202 produces autonomous frames 802 
from the input video signal and outputs the frames to the 
autonomous frame recoder 204 via line 220. The autono- 
mous frame generator 202 will be described in more detail 
with reference to FIG. 3. The autonomous frame recoder 204 
in turn processes the autonomous frames 802 including 
extracting processing parameters, extracting the video data 
and setting up or initializing a recoder to process the 
extracted video data. The autonomous frame recoder 204 
will be described in more detail with reference to FIG. 4. The 
autonomous frame recoder 204 outputs a bitstream that has 
been adjusted in the number of bits used to represent the data 
on line 124. 

Referring now to FIG. 3, a preferred embodiment for the 
autonomous frame generator 202 of the present invention is 
shown. The autonomous frame generator 202 preferably 
comprises an access unit generator 302, a header extraction 
unit 304, an auxiliary information updating unit 306, an 
autonomous frame header buffer 308, a payload transfer unit 
310, and an autonomous frame construction unit 312. The 
autonomous frame generator 202 is particularly advanta- 
geous because is receives a video bitstream and converts the 
bitstream into a plurality of autonomous frames that can be 
processed independently by the autonomous frame recoder 
204. For example, the autonomous frames can be processed 
one at a time in sequence as describe with reference to FIG. 
4 or many at a lime as will be described in other embodi- 
ments. The partitioning of the bitstream into autonomous 
frame that can be independently processed allows the frames 
to be processed in any number of ways, therefore making the 
present invention fully scaleable and flexible. 

The autonomous frame generator 202 has an input 
coupled to line 122 to receive transport packets, for 
example, in MPEG-2 format. The input of autonomous 
frame generator 202 is formed by the input of the access unit 
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generator 302. The access unit generator 302 receives the 
packets at the transport layer and converts them to access 
units as described above with reference to FIG. 8. In an 
exemplary embodiment, this could be the same as 
de-multiplexer 104. The packets on the transport layer are 
divided into a transport header and a transport payload to 
form a unit of convenient size for transmission over the 
channel. The packets on the transport layer are received by 
the access unit generator 302 and reformatted into access 
units including an AUH and AUP. The access unit generator 
302 uses the one or more transport headers to reformat the 
transport payloads into AU having an AUH and AUP, and 
then outputs the AU. The transport headers are discarded 
once the AUs have been generated. 

The output of the access unit generator 302 is provided as 
an input to both the header extraction unit 304 and the 
payload transfer unit 310. The payload transfer unit 310 is 
similar to the header extraction unit 304, but processes the 
AUs to generate the AUP. In the simplest embodiment, the 
payload transfer unit 310 receives an AU and identifies the 
picture data (as shown in Table 1) and outputs the picture 
data as the frame payload. For the case of MPEG-2 video 
bitstream, the frame payload is preferably a subsection of the 
MPEG-2 video bitstream, in the form of either elementary 
stream, PES stream, transport stream or program stream. 
The output of the payload transfer unit 310 is coupled to an 
input of the autonomous frame construction unit 312 to 
provide the AUPs for joining with the AFH to form an 
autonomous frame. 

The header extraction unit 304 receives the AU and 
extracts the AUH from the AU. The header extraction unit 
304 acquires the data from the AUH that is used to create the 
AFH. The AUH for MPEG-2 may have one of three formats. 
First, the AUH may include only the picture__header, the 
picture_extension and* the extension_user_data. In such 
case, the header extraction unit 304 determines these three 
values from the AUH and outputs them. Second, the AUH 
may include the GOP_Jieader, user_data, picture„header, 
the picture_extension and the extension__user_data. In this 
second case, the header extraction unit 304 determines these 
five values from the AUH and outputs them. Finally, the 
AUH may include a sequence_header, a sequence_ 
extension, sequence_user_data, the GOP_header, user_ 
data, picture_header, the picture_extension and the 
extension_user_data. In this case all the values are deter- 
mined and output by the header extraction unit 304. These 
values will be used in turn by the autonomous frame 
construction unit 312 to create the autonomous frame. Thus, 
the header extraction unit 304 provides the auxiliary infor- 
mation which for the case of MPEG-2 video bit stream, may 
include some or all of the following: a sequence header, a 
sequence extension, an extension and user data, a group of 
picture header, a picture header and a picture coding exten- 
sion. 

The output of the header extraction unit 304 is coupled to 
the input of the auxiliary information updating unit 306. The 
auxiliary information updating unit 306 also has a data and 
control outputs coupled to the autonomous frame header 
buffer 308. The auxiliary information updating unit 306 
maintains and keeps the autonomous frame header buffer 
308 current. As additional auxiliary information is extracted 
from each received AU by the header extraction unit 304, the 
auxiliary information updating unit 306 stores the new 
information al the autonomous frame header buffer 308. The 
auxiliary information updating unit 306 also receives signals 
from a recoding algorithm controller/selector indicating the 7 
recoding method used, the : desired bit budget for this picture 
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and other information not available from the current or AFHs and only process the AFPs. The recoder 408 then 

previous AUH, but needed for receding. The auxiliary outputs a bitstream which may be an elementary stream, 

information updating unit 306 is also responsible for man- packetized elementary stream (PES), transport, or an 

aging the data in autonomous frame header buffer 308 since autonomous frame. The controller 402 is coupled to control 

there may be auxiliary information for a number of streams 5 the parser 404, the initialization unit 406 and the recoder 408 

stored in the autonomous frame header buffer 308. to coordinate and control the operation of the above men- 

The autonomous frame header buffer 308 is any one of a tioned components. For example, the recoder 408 cannot 

variety of conventional buffers and is used to store the data start recoding until the parser 404 has completed parsing by 
necessary for re-encoding any given picture. Tfie^utonoT^ extracting and providing the auxiliary information and the 

molis~fr1une~he^^^^ initialization unit 406 has properly set the recoder* s states, 

^auxiliary information for a plurality of bitstreams/The Once the initialization is complete, the controller discon- 

iautonomous frame header buffer 308 preferably stores a nects the recoder 408 from the initialization unit 406 and 

current list of values for the sequence_header, a sequence_ performs the recoding using the input on 410. 

extension, sequence__user__data, the GOP_header, user_ The recoder 204 is particularly advantageous because it 

data, picture_header, the picture_extension and the 15 provides autonomous operation by virtue of fact that the 

extension„user_data for each stream. The autonomous recoder 408 can perform the operation completely indepen- 

frame header buffer 308 also stores other auxiliary informa- dently of the processing done on the previous and next 

tion required by a re-encoder to process the AFP. The access units. This capability allows several flexible imple- 

autonomous frame header buffer 308 is coupled to both the mentations of the recoding as will be described in detail 

auxiliary information updating unit 306 and the autonomous 20 below. Again, while the present invention is discussed in the 

frame construction unit 312. context of processing of MPEG-2 video elementary streams, 

The autonomous frame construction unil 312 is coupled to the same techniques, with minor modifications, can be 
the autonomous frame header buffer 308 to receive data for applied to the processing of MPEG-2 transport streams, 
generating the AFH and to the payload transfer unit 310 to MPEG-2 PES, MPEG-2 program streams, MPEG-1 streams, 
receive the data used to form the AFP. The autonomous 25 or H.26x based video conferencing signals. Other additional 
frame construction unit 312 basically packages the auxiliary information and protocol data units, such as MPEG-2 trans- 
information retrieved from the autonomous frame header port stream over ATM protocols, layered on top of the above 
buffer 308 into the autonomous frame header (AFH), once signals can also be handled similarly, 
per access unit. All information needed to describe the state For the present invention, recoding is defined in its 
of the recoder for it to perform recoding is retrieved and 30 broadest sense to include partial decoding, recoding, 
stored as the AFH. The autonomous frame construction unit re-quantization, re-transforming, and complete decoding 
312 then transfers the bitstream subsection of the AUP into and recoding. Referring now to FIG. 5, each of these type of 
the AFP. Thus, an autonomous frame is created and output recoding are defined with more particularity. FIG. 5 is used 
on line 220. to show various possibilities for the recoder 408. Some of 

Referring now to FIG. 4, a preferred embodiment for the 35 the elements shown may also be needed for decoding and 

autonomous frame recoder 204 of the present invention is encoding of the video data. Hence in actual implementation, 

shown. The autonomous frame recoder 204 comprises a these common elements may be shared between the recoder 

parser 404, an initialization unit 406 and a recoder 408 408 and the decoder/encoder. Partial decoding refers to path 

operating under the direction of a controller 402. The E where the bitstream is partially decoded, decode system 

autonomous frames are designed to allow autonomous 40 syntax, and video syntax down to the picture header to 

recoding. Specifically, the autonomous frame recoder 204 of perform frame accurate flexible splicing, Recoding refers to 

the present invention is one that can re-encode the access path D where variable length encoding and decoding are 

unit of compressed video elementary stream in an entirely performed and the DCT coefficients may be truncated to 

self-contained manner. In other words, the information con- zero without even going through the inverse quantization 

tained in the frame header of an autonomous frame is 45 steps. This approach requires the least processing, but in 

sufficient for the system to perform the re-encoding without general causes the greatest amount of quality degradation, 

any external input. The parser 404 has a data input, a data Re-quantization refers to path C where variable length 

output and a control input. The data input of the parser 404 encoding,, de-quantization, quantization and decoding are 

is coupled to line 220 to receive autonomous frames from performed but no transform coding is used. The transform 

the autonomous frame generator 202. The parser 404 50 coefficients (DCT coefficients) are requantized before VLC 

receives the autonomous frames, separates the AFH and encoded back. This is the approach preferably used for the 

AFP, and outputs corresponding AFHs and AFPs on line 410 recoder 408. Re-transformation refers to path B where 

such that each portion of the autonomous frame is identified variable length encoding, de-quantization, inverse 

for later processing. The output of the parser 404 is coupled transform, transform coding, quantization and decoding are 

to an input of the initialization unit 406 and a data input of 55 performed. The video frames are constructed without using 

the re-encoder 408. The initialization unit 406 identifies the motion compensation. In the case of B or P pictures, this 

AFH, and extracts the content in the frame header including would mean some of the coded blocks are motion estimated 

the auxiliary information and the AUH. This content is residual errors. Some form of spatial filtering may be used 

output by the initialization unit 406 and input to a control before forward transform coding is used in the encoding 

input of the re-encodcr 408. The content is used to initialize 60 process. Recoding refers to path A where the bitstreams arc 

the re-encoder 408. The re-encoder 408 is initialized, and as complete decoded to raw video and then encoded including 

a result of this initialization, the re-encoder 408 is set to a the use of motion compensation. Each of the paths A, B, C, 

state just as if it had completed the decoding of the last D, E includes a rate converter for adjusting the rate of the 

access unit prior to this access unit. The recoder 408 also has bitstream to ensure buffer compliance. Each of the rate 

the data input coupled to the output of the parser 404. The 65 converters may be different. For example, the rate converter 

recoder 408 performs the recoding processing using the AUP on path A may be a spatial filter and the rate converter on 

as input. At the data input, the recoder 408 disregards the path C may perform a quantization step size adjustment 
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while the rate converter on path D performs high frequency At this phase, the recoder 612 functions as if it has been 

elimination. Those skilled in the art will also recognize that processing a continuous flow of bitstream up to this point, 

the components of the recoder 408 used (e.g., the path with all parameters needed for the decoding updated, 

through the recoder 408) could also be variably controlled to Therefore, the key function of the controller 614 is to ensure 

provide variable bit rate conversion using the recoder 408. 5 the recoder 612 does not start processing before the header 

In various embodiments, the recoder 408 may include all, parser 606 and the initialization unit 610 have completed 

only some or any combination of these components accord- their processing of the autonomous frame header and have 

ing to which of recoding, re-quantization, re-transforming properly initialized the recoder 612. 

and recoding is performed. Referring now to FTC 7, a block diagram of a third 

More specifically, the recoder 408 preferably performs 10 embodiment for the autonomous frame processing unit 108c 
only requantization, and thus, follows path C through the of the present invention is shown. The third embodiment of 
rate converter. The inverse transform, motion compensation, the autonomous frame processing unit 108c preferably com- 
motion estimation, transform and additional rate converters prises a pre-parser 602, an autonomous frame creator 604, a 
on paths A, B, D, and E are therefore not required according header parser 606, a payload parser 608, an initialization 
this embodiment However, in alternate embodiments, the 15 unit 610, and a recoder 612. Many of the components of the 
recoder 408 may perform any level of recoding provided by third embodiment are similar to like components of the 
paths A through E. Generally, motion compensation is most second embodiment, and like reference numerals have been 
computationally expensive, transform coding and inverse used for like terms. The third embodiment for the autono- 
transform coding are also quite expensive. For example, in mous frame processing unit 108c is also similar to the first 
FIG. 5, if we take the path B, motion compensation is 20 embodiment in that the pre-parser 602 and the autonomous 
avoided. If we take path C, both motion compensation and frame creator 604 could be considered to be the autonomous 
transform coding are eliminated. If we take path D, quan- frame generator 202; and the header parser 606, the payload 
tization steps are also eliminated, in addition to motion parser 608, the initialization unit 610, and the recoder 612 
compensation and transform coding. Of course, path A could be considered the autonomous frame recoder 204. The 
performs the entire decoding and encoding processes are 25 third embodiment for the autonomous frame processing unit 
performed, resulting in the most flexibility and quality 108c also includes a controller 714 that is described in more 
potential, at the cost of being most expensive. detail below. 

Referring now to FIG. 6, a block diagram of a second As shown in FIG. 7, the pre-parser 602 has an input 

embodiment for the autonomous frame processing unit 108fo coupled to line 122 to sequentially receive a plurality of 

of the present invention is shown. The second embodiment 30 access units. The pre-parser 602, however, has a single 

of the autonomous frame processing unit 1086 preferably output coupled to the input of the autonomous frame creator 

comprises a pre-parser 602, a header parser 606, an initial- 604. The pre-parser 602 is used to generate the AFH. The 

ization unit 610, and a recoder 612. liie pre-parser 602 has pre-parser 602 extracts the AUH from the input streams on 

an input coupled to line 122 to sequentially receive a line 122 and combines the information in the AUH with 

plurality of access units. The pre-parser 602 has a plurality 35 other auxiliary information to form the AFH. The pre-parser 

of outputs coupled to line 620 and 622, respectively. The 602 then outputs the AFH to the autonomous frame creator 

pre-parser 602 is used to convert access units to autonomous 604. The autonomous frame creator 604 has a second input 

frames. The pre-parser 602 is used to generate a logical coupled to line 122 to receive the access units. The autono- 

autonomous frame and outputs the AFH on line 620 and the mous frame creator 604 also parse the access units, but it 

AFP on line 622. Line 620 is coupled to the header parser 40 extracts the AUP, and uses it to form the AFP. The autono- 

606. The header parser 606 extracts the content in the frame mous frame creator 604 then combines the AFP with the 

header including the auxiliary information and the AUH. AFH from the pre-parser 602 to produce autonomous frames . 

The header parser 606 outputs the auxiliary information and that are provided at the output of the autonomous frame 

the AUH to the initialization unit 610. The initialization unit creator 604. 

610 applies the auxiliary information and the AUH to the 45 The output of the autonomous frame creator 604 is 

recoder 612 to initialize the recoder 612. Again, the recoder coupled to the input of both the header parser 606 and the 

612 is set to a state just as if it had completed the decoding payload parser 606. The header parser 606 operates as has 

of the last access unit prior to this access unit The recoder been described above to extract the content in the frame 

612 is a similar type to that disclosed above with reference header including the auxiliary information and the AUH. 

to FIG. 5, 50 The header parser 606 outputs the auxiliary information and 

The controller 614 shown in FIG. 6 coordinates the the AUH to the initialization unit 610. The initialization unit 

operation of the pre-parser 602, the header parser 606, the 610 applies the auxiliary information and the AUH to the 

initialization unit 610, and the recoder 612. At the input of recoder 612 to initialize the recoder 612. Again, the recoder 

each new autonomous frame, the controller 614 first disables 612 is set to a state just as if it had completed the decoding 

the operation of the header parser 606, the initialization unit 55 of the last access unit prior to this access unit The recoder 

610, and the recoder 612. It then enables the pre-parser 602 612 is a similar type to that disclosed above with reference 

to generate the autonomous frame and separate the frame to FIG, 5. The payload parser 606 also receives the autono- 

into AFH and AFP. Next, the controller 614 enables the mous frames, however, the payload parser 606 extracts the 

header parser 606 to accept the AFH from the pre-parser AFP from each autonomous frame and provides the picture 

602. After that, the controller 614 sends the output of header 60 data of the AUP to the data input of the recoder 612. The 

parser 606 to the initialization unit 610. The controller 614 recoder 612 outputs an elementary bitstream that has been 

then disables the header parser 606 and enables the initial- recoded to the desired bit rate, whether it's CBR or VBR. 

ization unit 610. In this phase, the initialization unit 610 In FIG. 7, the header parser 606, the payload parser 608, 

resets the operation of recoder 612 and initializes it with the the initialization unit 610, and the recoder 612 are controlled 

values obtained from the header parser 606. In the last phase, 65 by controller 714. The controller 714 determines the 

the controller 614 enables the operation of recoder 612 and sequence of operation among all components, similar to the 

disables the header parser 606 and the initialization unit 610. controller 614 in FIG. 6. In this case, the sequence of 
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operation goes as follows. First, the controller 714 disables 
the payload parser 608, the initialization unit 610, and the 
recoder 612 and enables the header parser 606 to parse out 
the AFH. Second, the controller 714 then enables the ini- 
tialization unit 610 to use the parsed AFH to initial values 
used for decoding. Third, the controller 714 enables the 
operation of recoder 612. By this time, the recoder 612 
operates as if it is the middle of the on-going decoding .of a 
continuous stream. The controller 714 enables the payload 
parser 608 to allow it to parse out the AFP and output the 
result to the recoder 612. 

Referring now to FIG. 9, a block diagram of a fourth 
embodiment for the autonomous frame processing unit 108d 
of the present invention is shown. The fourth embodiment 
for the autonomous frame processing unit IQSd advanta- 
geously provides a divider 902, a plurality of autonomous 
frame generators 9040-904/1, a plurality of autonomous 
frame recoders 906a-906n and a combiner 908. The fourth 
embodiment for the autonomous frame processing unit 108d 
provides increased processing speed with a parallel archi- 
tecture at the cost of multiple autonomous frame generators 
904«-904/i and autonomous frame recoders 906^-906/1. 
The divider 902 has an input coupled to line 122 to receive 
access units. The divider 902 sequences or partitions the 
stream of access units received in a serial form and outputs 
a plurality of them in parallel. The divider 902 has a plurality 
of outputs, each output of the divider 902 is coupled to the 
input of a respective autonomous frame generator 
904a-904/i. For example, a first output of the divider 902 is 
coupled by line 922 to the input of autonomous frame 
generator 904a. It should be understood that each of the 
autonomous frame generators 904a-904n share a common 
buffer area (not shown) so that all the auxiliary information 
needed for the AFH is available to all other autonomous 
frame generators 904a-904/j. The outputs are coupled by 
other lines 922 to a respective autonomous frame generator 
904b-904/i. Each of the autonomous frame generators 
904fl-904/i is preferably similar to that described above with 
reference to FIGS. 2 and 3, and outputs autonomous frames. 
Each of the autonomous frame generators 904<z-904/i has an 
output that is coupled to a respective autonomous frame 
recoder 906a-906/t via line 924. The autonomous frame 
recoders 906a-906/j are preferably similar to that described 
above with reference to FIGS. 2 and 4. The output of each 
autonomous frame recoder 906o-906n is coupled by line 
926 to a respective input of the combiner 908. The combiner 
908 combines the outputs of multiple autonomous recoders 
906a-906/i into another video bit stream in the same order 
as that of the original bit stream. The output of the combiner 
908 is coupled to and provided on line 124. This architecture 
is advantageous since the access units from the same video 
elementary stream are first partitioned into autonomous 
frame data units and then distributed to different processors 
(autonomous recoders 906a-906/i) for recoding. This 
scheme allows parallel processing of a single compressed 
video bit stream. 

Referring now to FIG. 10, a block diagram of a fifth 
embodiment for the autonomous frame processing unit 108c 
of the present invention is shown. The fifth embodiment for 
the autonomous frame processing unit 108e is similar to the 
fourth embodiment 1084 however, it provides both parallel 
processing and selective processing. For 'convenience and 
ease of understanding, like components from FIG. 9 are 
provided with like reference numerals and have a similar 
functionality. The fifth embodiment for the autonomous 
frame processing unit 108e differs from the fourth embodi- 
ment by providing direct transfer lines between one or more 
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of the outputs of the divider 902 and respective inputs of the 
combiner 908. For example, this is illustrated in FIG. 10 by 
lines 1002 and 1004. These direct transfer lines 1002 and 
1004 allow the incoming video bitstream to be passed 
directly to the combiner 908 without recoding the access 
units transferred along these lines 1002 and 1004. The 
divider 902 provides access units on all its outputs. For the 
outputs connected to the combiner 908 by the direct transfer 
lines 1002 and 1004, the access units are passed to the 
combiner 908 unchanged. Thus, this embodiment provides 
selective recoding by virtue of selected access units being 
transferred along the transfer lines 1002 and 1004 while the 
other pass through autonomous frame recoders 906a-906/i. 
The combiner 908 combines the outputs of multiple autono- 
mous recoders and the direct transfers into another video bit 
stream in the same order as that of the original bit stream. 
The divider 902 and combiner 908 of FIGS. 9 and 10 operate 
in a coordinated manner. Using FIG. 9 as an example, the 
divider 902 first partitions the incoming (single) bitstream 
along access unit boundaries. If the input signal is not in 
elementary stream format, the divider 902 performs the 
equivalent operations on the associated transport packets, 
and in this case, the access units are encapsulated as the 
payload of the corresponding transport packets. 
Furthermore, the divider 908 delivers the resulting access 
units to the autonomous frame generators sequentially. The 
order at which 904a-904n receives the autonomous frames 
can be, for example, in round-robin order or in some other 
fixed order. The combiner 908, correspondingly performs 
the time sequential concatenation operation of the resulting 
outputs from the autonomous frame recoders 906a-906/i by 
following the same order as the divider 902. Therefore, some 
form of buffering (although not shown) may also be neces- 
sary between the combiner 908 and the autonomous frame 
recoders 906a-906n as will be understood by those of 
ordinary skill in the art. 

FIG. 11 shows a block diagram of a sixth embodiment of 
the autonomous frame processing unit 108/ of the present 
invention. The sixth embodiment of the autonomous frame 
processing unit 108/ provides selective recoding of a single 
video bitstream. Selective recoding refers to a process that 
recodes portions of a compressed video. Specifically, in the 
preferred embodiment, one access unit may be recoded, 
while the next one is not recoded. The autonomous frame 
processing unit 108/ preferably comprises an autonomous 
frame generator 1102, a first switch 1106, an autonomous 
frame recoder 1104, a second switch 1108, and a by-pass 
path 1110. The autonomous frame generator 1102 has an 
input and an output. The input of the autonomous frame 
generator 1102 is coupled to line 122 to receive a serial 
stream of access units. The autonomous frame generator 
1102 is similar to that described above with reference to 
FIGS. 2 and 3, and generates an autonomous frame for each 
access unit. The autonomous frames are output by the 
autonomous frame generator 1102 to the input of the first 
switch 1106. The construction of the autonomous frames 
ensures that the recoding can be done independently using 
only the autonomous frame as the input. The first switch 
1106 also has a first output and an second output. The first 
output of the first switch 1106 is coupled to the input of the 
autonomous frame recoder 1104 and the second output of 
the first switch 1106 is coupled to the by-pass path 1110. The 
first switch 1106 may be positioned so that it couples the 
output of the autonomous frame generator 1102 to the input 
of the autonomous frame recoder 1104, or the output of the 
autonomous frame generator 1102 to the by-pass path 1110. 
Thus, the autonomous frames are either recoded by the 
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autonomous frame recoder 1104 or directly passed to the 
output line 124. The autonomous frame recoder 1104 is 
similar to that described above with reference to FIGS. 2 and 
4, and recedes the autonomous frames. The output of the 
autonomous frame recoder 1104 is coupled to a first input of 
the second switch 1108. The second switch 1108 has a pair 
of inputs and an output, and may be positioned to output 
either the signal applied at the first input or the signal applied 
at the second input. The second input of the second switch 



statistical multiplexing on pre-compressed video signals 
without the use of real-time compression systems. The 
autonomous frame processing can also be used to perform 
statistical remultiplexing. In this case, the statistical remul- 
tiplexer performs recoding, selective or complete receding, 
on all of the participating channels so that the multiplexed 
output bit stream has a given constant rate. Statistical 
remultiplexing can be considered as the combination of 
selective recoding and channel hopping (described below). 



1108 is coupled to the by-pass path 1110. The output of the 10 On exemplary embodiment for a statistical re-multiplexer 



second switch 1108 is coupled to line 124 to provide the 
selectively recoded bitstream. The first and second switches 
1104 and 1108 are synchronized to be switched at the same 
time to be coupled either the autonomous frame recoder 
1104 or the by-pass path 1110. By selectively switching the 
positions of the first and second switches 1104 and 1108 
between bypass or recoding mode, selected autonomous 
frames may be recoded. The switches 1106 and 1108 are 
controlled by a separate controller, not shown in FIG. 11, but 



1200 using autonomous frame processing is shown in FIG. 
12 and comprises a plurality of pre-parsers 1202, a process 
allocator 1204, a plurality of autonomous frame recoders 
1206, a re-multiplexer 1208 and a controller 1210. 

The plurality of pre-parsers 1202 are similar to those 
described above with reference to FIGS. 6 and 7. The 
plurality of pre-parsers 1202 are each respectively coupled 
to receive a different video stream being statistically multi- 
plexed. The plurality of pre-parsers 1202 convert the access 



as will be understood by those of ordinary skill in the art. 20 units (or transport packets or PES packets that contain the 



The controller takes it's decision from algorithms related to 
the overall recoding needs of the output channel. For 
example, if the output channel is capable of handling the 
signal without any further recoding, the controller simply 
sets both switches to by-pass mode. If the output channel is 
running into severe bandwidth over usage, the controller 
may set the two switches to processing through the autono- 
mous frame recoder 1104 all the time. The more general 
situation is that the controller takes the decision to by-pass 
or process from a separate rate controller that operates to 
optimize the bandwidth utilization of the output channel by 
recoding some autonomous frames only when bandwidth 
utilization temporarily exceeds capacity. 

FIG. 12 is a block diagram of a first statistical 



25 



30 



access unit) into autonomous frames. If one pre-parser is 
dedicated to the pre -parsing of one channel as preferred, 
multiple pre-parsers are needed. Otherwise, the same pre- 
parser may be used, on a time division basis, to handle the 
pre-parsing of multiple channels of video channels. The 
outputs of the plurality of pre-parsers 1202 are coupled to 
respective inputs of the process allocator 1204. The process 
allocator 1204 assigns one of the plurality of autonomous 
frame recoders 1206 a given access unit of a given video bit 
stream, and its corresponding autonomous frame. The pro- 
cess allocator 1204 determines which autonomous frame is 
going to be processed by which autonomous recoder 1206. 
The process allocator 1204 implements resource sharing 
algorithms that attempts to achieve optimal quality under 



re-multiplexer 1200 using autonomous frame processing 35 certain constraints. For example, the algorithm may maxi- 



according to the present invention. Statistical multiplexing, 
in general, is a technique that simultaneously encodes mul- 
tiple channels of video signals and combines the resulting bit 
stream into a single bit stream for transmission. Statistical 
multiplexing explores the variable rate nature of the com- 
pressed video bit streams and the statistical nature of such bit 
rates. Specifically, it combines multiple channels of variable 
bit rate (VBR) bit streams of compressed video so that the 
resulting multiplex has a constant fixed rate. This technique, 



mize the number of autonomous frame recoders in use at any 
time when the bandwidth utilization at the output of the 
re-multiplexer exceeds the channel capacity. The algorithm 
may also allocate the autonomous frame recoders according 
to user-set priorities on different video channels by reducing 
or by-passing recoding activities on those higher priority 
(namely requires less recoding) channels. A variety of algo- 
rithms for implementing statistical multiplexing are known 
in the art, any one of which may be used to control the 



when properly implemented, results in significant bandwidth 45 operation of the process allocator 1204. The process allo- 

savings when transmitting multiple channels of compressed , cator 1204 has a plurality of outputs each coupled to a 

digital video signals. The statistical multiplexing scheme j respective one of the plurality of autonomous frame recoders 

includes the following key components: 1) analysis of video 1206, and for delivering autonomous frames. The plurality 

^contents to determine the optimal bit budget allocated toy of autonomous frame recoders 1206 are similar to those that 

<each coded picture; 2) coded picture type decision: whether 50 have been described above, and can independently recode 



to use motion compensation; 3) coordinated selection of bit 
( rates across all channels to result in total fixed rate; 4) rate 
\ control or coding of pictures in each channel that meets the 
( ; bit usage budget determined by the step 3; 5) monitoring and 
compensation of timing information on all channels 
, (required to carry real-time compressed video signals); 6) 
J scheduling and multiplexing of different channels into a 
fixed bandwidth channels * — - ■ 

Statistical remultiplexing is a technique that simulta 



autonomous frames. Each of the plurality of autonomous 
frame recoders 1206 has an output coupled to a respective 
input of the re-multiplexer 1208. The re-multiplexer 1208 is 
preferably an MPEG- 2 transport packet re-multiplexer that 
55 accepts multiple data paths, each carrying a stream of 
/ transport packets (or PES packets or simply video elemen- 
tary stream) which contains one compressed video bit 
stream and possibly one or more audio or data packets. 
Finally, a controller 1210 is coupled to the plurality of 



neously recodes multiple channels of video signals and 60 pre-parsers 1202, the process allocator 1204, the plurality of 

combines the resulting bit streams into a single bit stream for autonomous frame recoders 1206, and the re-multiplexer 

transmission. The key difference from statistical 1208. The controller 1210 maintains a database of inform a- 

multiplexing, therefore, is that the inputs to the statistical ^ tion from the pre-parser 1202, process allocator 1204, 

remultip lexer are pre-compressed bit streams. The recoding, . autonomous recoder 1206 and re-multiplexer 1208. The 

together with the rate control and remultiplexing performs 65 information may include, but is not necessarily limited to, 

the same functions as that of statistical multiplexing. Sta- /bii _usage per coded picture before' and after the recoding, - 

tistical remultiplexing allows the system operator to perform ^ state of the processor allocator, _buffer conditions in the~ 
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fre^multiplexer, etc. Therefore, the controller 1210 manages 
Hhe overall operation of the statistical re -multiplexing. It is 
important here to understand the relationship between the 
process allocator 1204 and the plurality of autonomous 
frame re code is 1206. The process allocator 1204 connects a 
given autonomous frame from one video channel to one of 
the recoder resources. The connection pattern is not a static 
one. It is dynamically reconfigured from time to time, 
depending on the overall statistical multiplexing needs and 
rate control operation. 

FIG. 13 is a block diagram of a second statistical 
re-multiplexer 1300 using autonomous frame processing 
according to the present invention. The second statistical 
re-multiplexer 1300 is used to process multiple channels, 
and allows the autonomous frame processing to channel hop 
between any of the input channels. While FIG. 13 illustrates 
only five channels and a single autonomous frame generator 
and recoder pair, those skilled in the art will recognize that 
this embodiment could be adapted to process any number of 
channels and additional pairs of autonomous frame genera- 
tors and recoders could be provided. The channel hopping 
implementation uses the autonomous frame construct to 
recoder multiple channels of video bit streams. Specifically, 
a single recoder can be used to recode the autonomous 
frames from multiple channels of video signals, provided 
that the recoder has sufficient processing power to handle the 
tasks. 

The channel hopping implementation of the second sta- 
tistical re-multiplexer 1300 preferably comprises a first 
crossbar switch 1302, an autonomous frame generator 1304, 
an autonomous frame recoder 1306, a second crossbar 
switch 1308 and a controller 1310. In FIG. 13, five video 
channels are being receded by the same autonomous frame 
generator 1304 and autonomous frame recoder 1306. The 
access units of the five channels input to the first crossbar 
switch 1302 may or may not be time aligned. The time 
required to process each of the access units usually varies 
from frame to frame. Using the autonomous frame generator 
1304 and the autonomous frame recoder 1306 it is possible 
to recode one access unit in one channel and then recode the 
access unit of another channel. The autonomous frame 
generator 1304 and the autonomous frame recoder 1306 can 
either be used to perform the receding of all channels or 
receding of selected access units of all channels, using the 
approach shown in FIG. 13. The autonomous frame genera- 
tor 1304 is of the type described above and converts access 
units into autonomous frames. The autonomous frame 
recoder 1306 is of the type described above and performs 
receding to produce access units or transport packets con- 
taining those access units. The first crossbar switch 1302 
preferably has a plurality of inputs and an plurality of 
outputs. The first crossbar switch 1302 is an input switch 
device that can select the input of the autonomous frame 
generator 1304 to accept access units (or transport packets) 
from any given video channel. Similarly, the second cross- 
bar switch 1308 has a plurality of inputs and an plurality of 
outputs, and is an output switch device that is synchronized 
with the first crossbar switch 1302 such that the autonomous 
frame recoder 1306 is always on the same channel as the 
input selected by the autonomous frame generator 1304. In 
addition, the first and second crossbar switches 1302, 1308 
have the capability to delineate input signals along the 
access unit boundaries. This is not shown in the figure, but 
should be assumed as a function of first crossbar switch 
1302. The second crossbar switch 1308 performs the reverse 
function of the first crossbar switch 1302 correspondingly. 
As shown, this embodiment also includes a plurality of 
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direct transfer paths 1330, 1332, 1334, and 1336 that directly 
transfer the input video channel if the channel is not selected 
for input to the autonomous frame generator 1304. Finally, 
this embodiment includes a controller 1310 coupled to 

s control the first and second cross-bar switches 1302, 1308. 
The controller coordinates the operation to make sure that 
the first and second crossbar switches operate in sync, i.e., 
the processed output from chl is sent to the input of crossbar 
switch associated with chl, so on and so forth. 

While the present invention has been described with 
reference to certain preferred embodiments, those skilled in 
the art will recognize that various modifications may be 
provided. These and other variations upon and modifications 
to the preferred embodiments are provided for by the present 
invention which is limited only by the following claims. 

15 What is claimed is: 

1. A system for transcoding compressed video streams 
using a self-contained data unit processing video data, the 
system comprising: 

a self-contained data unit generator having an input and 
20 an output for receiving a stream of video data and 
converting the stream of video data into a plurality of 
self-contained data units, the input of the self-contained 
data unit generator coupled to receive compressed 
video streams; and 
25 a self-contained data unit recoder having an input and an 
output for retrieving receding parameters, initializing a 
recoder and receding video data included in the self- 
contained data units, the input of the self-contained 
data unit recoder coupled to the output of the self- 
30 contained data unit generator. 

2. The system of claim 1 wherein the self-contained data 
unit generator divides the video stream into a plurality of 
self-contained data units, each of the self-contained data 
units having a header portion and a pay Load portion. 

35 3. The system of claim 2 wherein each of the self- 
contained data units is an autonomous frame, and an autono- 
mous frame is comprised of: 

a frame header including all header information from the 
original video data plus additional information to allow 
the frame to be receded, and 
a frame payload including the original video data infor- 
mation. 

4. The system of claim 1, the self-contained data unit 
45 generator further comprises: 

a header extraction unit having an input and an output for 
extracting header information from the video stream, 
the input of the header extraction unit coupled to 
receive the stream of video data; 

50 an auxiliary information updating unit having an input 
and an output for generating the recoding parameters, 
the input of the auxiliary information updating unit 
coupled to the output of the header extraction unit; 
a payload transfer unit having an input and an output for 

ss receiving and identifying picture data and generating a 
frame payload, the input of the payload transfer unit 
coupled to receive the stream of video data; and 
an autonomous frame construction unit having a first 
input, a second input and an output for producing an 

60 autonomous frame, the first input of the autonomous 
frame construction unit coupled to the output of the 
auxiliary information updating unit to receive the frame 
header, and the second input of the autonomous frame 
construction unit coupled to the output of the payload 

65 transfer unit for receiving the frame payload. 

5. The system of claim 4, wherein the self-contained data 
unit generator further comprises an access unit generator 
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having an input and an output for receiving packets at the 
transport layer and converting them to access units, the input 
of the access unit generator coupled to receive the stream of 
compressed video data, and the. output of the access unit 
generator coupled to the input of the header extraction unit 
and the payload transfer unit. 

6. Ihe-system of claim 4, wherein the self-contained data 
unit generator further comprises an autonomous frame 
header buffer having an input and an output for storing to 
store the data necessary for re -encoding any given picture, 
the input of the autonomous frame header buffer coupled to 
the output of the header extraction unit, and the output of the 
autonomous frame header buffer coupled to the input of the 
autonomous frame construction unit. 

7. The system of claim 4, wherein the header extraction 
unit acquires a plurality from the group of picture_header, 
the picture_extension, the extension_user_data, and user_ 
data out of the video data stream. 

8. The system of claim 1, the self-contained data unit 
recoder further comprises: 

a parser having an input and an output for identifying a 
header portion from the self-contained data unit for 
further processing, the input coupled to the output of 
the self-contained data unit generator for receiving 
self-contained data unit, the output of the parser pro- 
viding the identified header portions; 

an initialization unit having an input and an output for 
receiving recoding parameters from the header portions 
identified by the parser and providing the recoding 
parameters as control outputs, the input of the initial- 
ization unit coupled to the output of the parser; 

a recoder unit having an first, second and third inputs and 
an output for modifying the bit rate of a video stream, 
the first input of the recoder coupled to receive picture 
data, the second input, of the recoder coupled to the 
output of the initialization unit to receive recoding 
parameters; and 

a controller for controlling the processing of self- 
contained data units, the controller coupled to the 
parser, the initialization unit, and the third input of the 
recoder for controlling the recoding process. 

9. The system of claim 8, wherein the self-contained data 
unit recoder further comprises a pre-parser having an input, 
a first output and a second output for separating the header 
portion and the payload portion of each self-contained data 
unit, the input of the pre-parser coupled to the output of the 
self-contained data unit generator for receiving self- 
contained data units, the first output of the pre-parser 
coupled to the input of the parser, the second output of the 
pre-parser. coupled to the first input of the recoder. 

10. The system of claim 8, wherein the self-contained data 
unit recoder further comprises a second parser having an 
input and an output for separating the payload portion of 
each self-contained data unit, the input of the second parser 
coupled to the output of the self-contained data unit gen- 
erator for receiving self-contained data units, the output of 
the second parser coupled to the input of the recoder. 

U. The system of claim 1, wherein the system further 
comprises: 

a second self-contained data unit generator having an 
input and an output for receiving a second stream of 
video data and converting the second stream of video 
data into a plurality of self-contained data units, the 
input of the second self-contained data unit generator 
coupled to receive compressed video streams; and 

a second self-contained data unit recoder having an input 
and an output for retrieving recoding parameters, ini- 
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tializing a recoder and recoding video data, the input of 
the second self-contained data unit recoder coupled to 
the output of the second self-contained data unit gen- 
erator. 

12. The system of claim 11, further comprising: 

a divider having an input and a plurality of outputs, for 
sequences and partitions the stream of video data, the 
input of the divider coupled to receive the stream of 
video data, a first output of the divider coupled to the 
input of the self-contained data unit generator, and a 
second output of the divider coupled to the input of the 
second self-contained data unit generator; and 
a combiner having a plurality of inputs and an output, 
for combining a plurality of bitstreams, the output of 
the divider coupled to provide a stream of video data, 
a first input of the divider coupled to the output of the 
self-contained data unit recoder, and a second input 
of the divider coupled to the output of the second 
self-contained data unit recoder. 

13. The system of claim 12, further comprising: 

a plurality of additional pairs of a self-contained data unit 
generator and a self-contained data unit recoder serially 
coupled, each of the pair allowing the recoding of video 
data, and 

wherein the divider has a respective output coupled to the 

input of one additional pair; and 
wherein the combiner has a respective input coupled to 

the output of one additional pair. 

14. The system of claim 12, further comprising at least 
one signal line coupled between a respective output of the 
divider and a respective input of the combiner. 

15. The system of claim 1, further comprising: 
a by-pass signal line; 

a first switch coupled to the output of the self-contained 
data unit generator, the by-pass signal line and the input 
of the self-contained data unit recoder, the first switch 
coupling the output of the self-contained data unit 
generator to either the by-pass signal line or the input 
of the self-contained data unit recoder; and 

a second switch coupled to the output of the self- 
contained data unit recoder, the by-pass signal line and 
an output signal line, the second switch coupling either 
the output of the self-contained data unit recoder or the 
by-pass signal line to the output signal line. 

16. The system of claim 11, further comprising: 

a first cross bar switch having an plurality of inputs and 
a plurality of outputs, for coupling one of the inputs to 
a respective output in response to a first control signal, 
a first output of the first cross bar switch coupled to the 
input of the self-contained data unit generator, and a 
second output of the first cross bar switch coupled to 
the input of the second self-contained data unit gen- 
erator; and 

a second cross bar switch having an plurality of inputs and 
a plurality of outputs, for coupling one of the inputs to 
a respective output in response to a second control 
signal, a first input of the second cross bar switch" 
coupled to the output of the self-contained data unit 
recoder, and a second input of the second cross bar 
switch coupled to the output of the second self- 
contained data unit recoder. 

17. A method for processing video data, the method 
including the steps of: 

receiving a video bitstream; 

dividing the video bitstream into a plurality of self- 
contained data units, each self-contained data unit 
including a portion of the video bitstream and recoding 
information; 
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processing the plurality of self-contained data units using 
only information contained therein, wherein processing 
comprises receding video data in the self-contained 
data unit. 

18. The method of claim 17 wherein the step of dividing 
the video bitstream into a plurality of self-contained data 
units, further comprises the steps of: 

extracting a plurality of frame headers including header 
information from the original video bitstream and 

extracting a plurality of frame payloads including the 
original video data information. 

19. The method of claim 18 wherein the step of dividing 
the video bitstream into a plurality of self-contained data 
units, further comprises the step of: updating information in 
the plurality of frame headers with the current state of a 
recoder. 

20. The method of claim 19 wherein the step of dividing 
the video bitstream into a plurality of self-contained data 
units, includes the step of temporarily storing the plurality of 
frame headers. 

21. The method of claim 19 wherein the step of dividing 
the video bitstream into a plurality of self-contained data 
units, includes the step of generating a self-contained data 
unit by combining an extracted frame header with updated 
auxiliary information and an extracted payload. 
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22. The method of claim 17 wherein processing the 
plurality of self-contained data units further comprises: 

parsing a self-contained data unit into a header portion 

and a payload portion; and 
initializing recoding using the header portion. 

23. The method of claim 22 wherein the step of parsing is 
performed by a first parser that parses a plurality of self- 
contained data units and identifies and outputs the header 
portion of each self-contained data unit, and a second parsers 

1 that parses the plurality of self-contained data units and 
identifies and' outputs the payload portion of each self- 
contained data units. 

24. The method of claim 17 wherein at least two of the 
self-contained data units are processed at the same time. 

25. The method of claim 24 wherein a first plurality of 
self-contained data units are not processed while the two of 
the self-contained data units are processed at the same time. 

26. The method of claim 17 wherein at least two of the 
self-contained data units are processed by different autono- 

1 mous recoding units. 

27. The method of claim 17 wherein a first self-contained 
data unit is processed and a second self-contained data unit 
is not processed. 

* * * * * 
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